Computer controlled microscope

ABSTRACT

A computer controlled microscope is provided with improved usability, which allows a user to quickly build, modify and reuse complex sample illumination and/or observation processes by introducing a recordings hierarchy in which a recording can be both a parent recording of one or more child recordings and a child recording to a single parent recording. A recordings hierarchy allows control parameter data to be inherited from a parent recording to a child recording. The group of child recordings linked to a parent recording is called a recordings collection. Viewed together, recordings and recordings collections form a tree-like hierarchy. Single load and store functions associated with a recording permit the entire recording and child recordings hierarchy to be stored on a storage device and reloaded into the computer of a computer controlled microscope.

Computer controlled microscopes allow users to set various parametersvia a user interface and to initiate the acquisition of image data onthe basis of parameter sets. The image data and the associated set ofparameters can be stored on a storage device, for example a computerhard disc or a network server.

In the following, a set of control parameter data, for example size andposition of an imaging area, resolution, illumination intensity,detection sensitivity and time data, will be called a recording. Arecording defines a sequence of operations and/or the status of amicroscope and can be loaded and used to configure the microscope,enabling, for example, the acquisition of image data under identicalconditions, in comparison with the acquisition of image data using thesame set of control parameter data before in a different location or ata later time.

In general, a microscope can be used to perform sample illuminationand/or sample observation. A recording defines both sample illuminationand/or sample observation by defining the operational status of themicroscope. Consequently, the execution of a recording is to beunderstood as the process of sample illumination or the process ofsample observation, as well as the combination of a plurality of thesetwo processes.

The recent trend in the life sciences away from the observation of“fixed” samples, towards the observation of live specimens requiresobservations at discrete time points over an extended period of time,maybe days. Specimen preparation usually results in a coverslip's worthof cells covering an area far wider than may be observed by a singlerecording. Observation of a single cell at discrete time points overseveral hours means the microscope is actually working for only a smallamount of the total experiment time, and that the microscope is missingthe chance to observe many other cells, a serious under utilization ofsample, microscope and scientists. Moreover, manual input over such longexperiment times can easily result in simple user errors.

It is an object of the present invention to provide a computercontrolled microscope with improved usability, which allows a user toquickly build, modify and reuse complex sample illumination and/orobservation processes.

This object is achieved by introducing a recordings hierarchy in which arecording can be both a parent recording of one or more child recordingsand a child recording to a single parent recording. A recordingshierarchy according to the present invention allows control parameterdata to be inherited from a parent recording to a child recording. Thegroup of child recordings linked to a parent recording is called arecordings collection. Viewed together, recordings and recordingscollections form a tree-like hierarchy. Single load and store functionsassociated with a recording permit the entire recording and childrecordings hierarchy to be stored on a storage device and reloaded intothe computer of a computer controlled microscope. In other words,initiating a save function for a single recording causes not only theroot recording, but all child recordings and recordings collections tobe saved also to the storage device. Initiation of a load operationcauses a root recording (i.e. a recording that has no parent recording)and child recordings and recordings collections to be created in thecomputer memory of the computer of the computer controlled microscope.

Each recordings collection may contain functions enabling recordings(and implicitly their child recordings) to be added to, removed from,and reordered within the collection, as well as functions enabling therecordings hierarchy to be traversed.

The standard microscope execute function is enhanced to allow therecordings hierarchy to be worked through with a single function call,passing as a parameter the highest level recording in the hierarchy,which is to be executed. In other words, a specific function “recordingexecute” first executes tasks specified by its own parameters, thenloops through all recordings contained within its collection of childrecordings, calling the same “recording execute” function on each ofthese recordings. Furthermore, any recording may be enabled or disabledwith respect to the “recording execute” function, i.e. may be markedsuch that the “recording execute” function of the microscope will onlyinitiate execution of those recordings which are enabled. The enablingor disabling of a recording may be achieved by introducing an “executionenabled” indicator as a further parameter of the recording.

Image data produced by each recording can be displayed in either aseparate window, or the same window for a whole recordings hierarchy.

The recording execute function is tolerant of child recordings in whichsome parameters or groups of parameters are undefined. These recordingsthen inherit the undefined parameters from parent recordings, forexample, either by copying the respective parameter values of the parentrecording or by referring to the respective parameter value of theparent recording.

It should be noted that a computer controlled microscope may comprise acomputer to control its settings and/or operations. A computercontrolled microscope may also be linked to an external computercontrolling its settings and/or-operations, for example a personalcomputer, or to a network of computers, which need not be located nextto the microscope. Further, it should be noted that the term“microscope” relates not only to a microscope as such but also to anyauxiliary device linked to or cooperating with the microscope in theprocess of image acquisition and specimen or sample handling. Thesedevices may include but are not limited to heating or cooling units, gasor liquid supply units, power supply units, sample manipulators etc.

In the following, the invention will be described in greater detail withreference to the drawings in which

FIG. 1 shows a schematic diagram of a computer controlled microscopeaccording to the invention;

FIG. 2 shows the basic structure of a recording according to theinvention;

FIG. 3 shows an example of an recording hierarchy according to theinvention;

FIG. 4 shows an example of a user interface of a computer controlledmicroscope according to the invention; and

FIG. 5 shows another example of a user interface of a computercontrolled microscope according to the invention.

In FIG. 1, components of a computer controlled microscope areschematically shown and will be described in some detail in order tofacilitate the understanding of the invention.

The computer controlled microscope comprises a stage 1 on which a samplecan be placed and which can be moved in a plane indicated by X and Y.The sample is illuminated by means of an illuminating device 3. Animaging device 4 is located such that an image of the sample can beacquired, i.e. such that the sample 2 is placed within the imaging area5 of the imaging device 4. The illumination device 3 and the imagingdevice 4 are located in the same housing as shown in FIG. 1 or may beprovided as separate devices. In the microscope shown in FIG. 1, stage1, illumination device 3 and the imaging device 4 are linked to acontroller 6, preferably a computer. The controller 6 controls theoperation and/or status of the microscope, i.e. the positioning device1, the illuminating device 3, and the imaging device 4 shown in FIG. 1.In order to control the microscope a control program is executed bycomputer 6 after having been loaded into the memory (not shown) ofcomputer 6. The control program is permanently stored and loaded from astorage device 7.

The user of the computer controlled microscope interacts with the systemthrough input devices like keyboard device 8, pointing device 9, forexample a computer mouse, a trackball, a touch screen or the like, and amicrophone 10 and through output devices like display 11 of which morethan one, as shown in FIG. 1, may be linked to the computer 6. On adisplay surface 12 of output device 11, a picture of sample 2 asrecorded by imaging device 4 is displayed for inspection by the user andin addition to control elements of the control program executed bycomputer 6.

Computer 6 controls the operation and/or status of the microscope, forexample by controlling the position of the positioning device 1, bycontrolling the kind and intensity of the illumination provided byilluminating device 3 and by controlling the shape, size and position ofthe imaging area 5 or imaging region 5 a grasped by imaging device 4.Further computer 6 receives image data from imaging apparatus 4 anddisplays the image of sample 2 on output device 11 and/or stores theimage on storage devices 7.

As mentioned before, the above description of a computer controlledmicroscope as schematically shown in FIG. 1 is given only to facilitatea better understanding of the invention described further below but isnot intended to limit the scope of the invention to a computercontrolled imaging device as shown in FIG. 1. However, the computercontrolled imaging device as shown in FIG. 1 clearly indicates that aplurality of parameters of the imaging device are controlled by computer6. These parameters include but are not limited to shape, size,orientation and position of the imaging region, resolution of theimaging device, illumination intensity, observation sensitivity, andtime data (like start-time and duration) etc.

The major improvement provided by the invention is achieved byintroducing a recording hierarchy into the way the computer works i.e.processes and stores/retrieves parameter data. The invention provides acomputer controlled microscope with additional functionality and,therefore expands its usability as a technical scientific instrument.

In FIG. 2 the structure of a single recording according to the inventionis shown, which may both a parent recording and a child recording in arecording hierarchy described in greater detail further below. As can beseen in FIG. 2, the recording comprises a set of parameters, for exampleshape, size and position of an imaging region, resolution, illuminationintensity, detection sensitivity, time data (like start-time andduration) and other data related to the process of sample illuminationor sample observation (image acquisition) or plurality of theseprocesses.

It should be noted that a recording may comprise a plurality of eachkind of parameter data. For example, a recording may comprise two ormore parameters defining different imaging regions with respect toshape, size and position. Also, a recording may comprise differentillumination and/or detection parameters.

Furthermore, a recording may comprise, as an additional parameter, anindicator as to whether or not the recording will be executed if a“recording execute” function of the computer controlled microscope isinitiated. If set, the execution indicator will cause the “recordingexecute” function of the microscope to also execute the specificrecording of which it is a parameter.

In FIG. 3 an example of a recordings hierarchy is shown. Each of therecordings in this exemplary hierarchy comprises a set of parameters asdiscussed with respect to FIG. 2. However, it is obvious to the personskilled in the art that recordings within a recordings hierarchyaccording to the invention may comprise additional parameters or may belimited to fewer or different parameters.

A root recording 0.0.0 as shown in FIG. 3 is characterized in that ithas no parent recording. The root recording 0.0.0 comprises controlparameter data such as the shape, the size and the position of animaging region, resolution of an imaging operation, illuminationintensity, detection sensitivity and time. In the example of FIG. 3 thisset of control parameter data is used for each recording shown in thefigure.

The root recording 0.0.0 has two child recordings, i.e. recording 1.0.0and 2.0.0, both of which comprise the set of control parameter data asmentioned above. Recording 1.0.0 is a parent recording to recordings1.1.0 and 1.2.0, of which recording 1.2.0 is a parent recording forrecording 1.2.1. Similarly, recording 2.1.0 is a child recording ofrecording 2.0.0 on one hand and a parent recording for recording 2.1.1on the other hand.

It is apparent to a person skilled in the art that this hierarchy can beextended by adding further recordings being child recordings to any oneof the recordings already being or becoming a part of the hierarchyshown in FIG. 3.

According to the invention, the control program of a computer controlledmicroscope comprises a function for creating a recording hierarchy byallowing the creation of a root recording and adding further recordingsas child recordings of the root recording or of child recordings createdin a previous step. The step of creating child recordings may also beunderstood as a function of adding recordings to the hierarchy. In apreferred embodiment, the control program of the computer controlledmicroscope further comprises the function of deleting a recording and/orthe function of reordering the recordings in the recording hierarchy.

A benefit of introducing a recording hierarchy into the control programof a computer controlled microscope is that it introduces thepossibility to inherit control parameter data from a parent recording toa child recording or group of child recordings.

For example, if a user has defined in a first step the root recording0.0.0 shown in FIG. 3 control parameter data like shape, size andposition of the imaging area, resolution of the image acquisitionprocess, illumination intensity, detection sensitivity and time havebeen defined in order to fully describe the status and/or operation ofthe computer controlled microscope. If the user defines in a second stepchild recording 1.0.0 at least some of the controlled parameter datapreviously defined or set for root recording 0.0.0 are used for thedefinition of the control parameter data of child recording data 1.0.0.Similarly, control parameter data of the root recording 0.0.0 isinherited during the creation of child recording 2.0.0.

Of course, the control program of the computer controlled microscopeaccording to the invention allows the user to override inherited controlparameter data, for example by redefining the shape, size and/orposition of the imaging region. With or without limitations based on thecontrol parameter data of the parent recording, the user may change thecontrol parameter data of the child recording.

The user may for example reduce the size of the imaging region of achild recording on one hand and increase the resolution of the imagingoperation on the other hand.

Preferably, when the user creates a further child recording, the controlparameter data of the parent recording are initially inherited by thechild recording created.

The user gains access to the improved technical functions of thecomputer controlled microscope via a graphical user interface displayedon an output device. In the following, an example of such a userinterface will be described. Obviously, the user interface may have adifferent look and arrangement of information.

In FIG. 4 an example of a graphical user interface is shown, which isdisplayed on the display device (see FIG. 1) of the computer controlledmicroscope, and which not only displays the image 13 of the sample 2under observation, but also comprises graphical control elements 14enabling the user to control the execution of the control program in thecomputer of the microscope.

The example in FIG. 4 shows the overview image 13, as defined by a rootrecording according to the invention, and one smaller imaging region 15defined by a respective child recording according to the invention. Theuser may define the shape, the size, the orientation and the position ofthe smaller imaging region by means of the pointing device (see FIG. 1),for example a computer mouse, in that he selects a draw-tool from thepalette 14 and “draws” the imaging region within the boundaries of theoverview image 13, i.e. the root recording. This technique is basicallyknown from other computer applications and is adopted for the inventionto allow a facilitated creation of a child recording.

In FIG. 5 another example of a graphical user interface is shown, whichis displayed on the display device (see FIG. 1) of the computercontrolled microscope, and which shows the recording hierarchy 16 in auser friendly format as well as graphical control elements 17 enablingthe user to control the execution of the control program in the computerof the microscope. The diagram 16 represents the recordings hierarchy. Aroot recording 18 and a single child recording 19 are shown as anexample of a recordings hierarchy according to the invention. Basicallythe structure of the hierarchy shown in FIG. 3 can be easily identifiedin the diagram of FIG. 5.

In the following some usage examples of the invention will be describedto give a better understanding of the scope of the invention and theadvantages achieved.

Example 1

Observation of a Field of Cells Spread Over a Wide Area

Whilst continuously scanning the imaging area with the microscope (i.e.continuously executing the “recording execute” function), the uservaries the scan stage position and recording parameters until a singlecell of interest is found. Once found, other parameters (for example,illumination intensity and detection sensitivity) can be adjusted untilan optimal image is obtained. A new imaging window is created, leavingthe image of the cell (and associated parameters) on the computer screenin the old window. The user repeats the process of finding cells. Whenan appropriate number of cells have been found, the user selects eachimaging window in turn, and presses a button on the window control bar,which causes the recording contained within the window to be added tothe root recording. The root recording thus will contain a number ofrecordings.

When data acquisition is restarted, each recording in the root recordingwill be scanned in turn, reusing the scan windows that were used to findthe cells.

The recording collection can be scanned at timed intervals, so that manyspatially remote cells may be observed time-multiplexed.

Example 2

Observation of a Field of Cells Spread Over a Small Area

A low magnification scan is made, creating an image in which a largenumber of cells can be identified. Using draw tools selected from a scanwindow control menu, cells of interest can be drawn on the display, thedrawn regions automatically defining child recordings, which are zoomedversions of the main recording, to which they are added. Initiation ofimage acquisition (recording execution) causes the root (overview)recording, and then all child (cell) recordings to be scanned insequence.

Acquisition of any one of the recordings can be disabled, for example,such that only the cells are scanned and not the overview recording. Thegraphical objects defining the cells in the overview window can bemanipulated during a scan to adjust the size parameters of the childrecordings as they are scanned.

Example 3

Combination of Wide Area and Small Area Observation

A basic recording comprising child recordings can be added to anotherrecording. Thus a recordings hierarchy can be built containing clustersof recordings, within each of which the scan stage doesn't have to bemoved (e.g. since this is slow).

Example 4

Optimal Observation of Different Regions within a Single Cell

Cells marked with fluorescent probes may have some structures containinga high density of fluorophore, and other structures with lowerdensities. An initial, non-optimal scan can be made, upon which theareas to be optimised are drawn, the drawn regions automaticallydefining child recordings, which are unzoomed versions of the rootrecording. Each child recording, by default inherits parameters (otherthan size) from the parent recording. Thereby, the user can optimiseillumination intensity and detection sensitivity for each region.

Example 5

Combination of Three-dimensional, Two-dimensional and One-dimensionalData Acquisition

A confocal fluorescence microscope is able to measure fluorescenceintensity within a three-dimensional volume. A single recording mightconstitute a single sample or line. A two-dimensional confocal image canrepresent a slice through a sample. A stack of slices thus constitutes athree-dimensional data representation of the sample. For such a machine,each recording within the recordings hierarchy is defined in threedimensions. A recordings hierarchy may therefore be constructed, whichcontains recordings of different dimensionality. Examples might include:

-   -   An overview two-dimensional slice through a field of cells,        containing child recordings, which are three-dimensional stacks.    -   A recording creating a single, overview two-dimensional image of        a single cell (Nomarski contrast), and a child recording, which        defines stack of fluorescent images.    -   A root recording might define a slice in one direction through a        sample, e.g. in a plane orthogonal to the optical axis. Drawing        a line onto the overview image causes a child recording to be        created. The extent of such a child recording can be extended so        that it constitutes a plane orthogonal to its parent

Example 6

“Illumination-only” Processes

There are many instances in which only illumination of a sample, and notobservation is interesting:

a) Photobleaching

A common technique in cell biology for observing cargo movements withcells is photobleaching. A cell containing proteins of interest markedwith fluorescent dyes is prepared. An initial image of the cell isacquired, after which a high power laser scan is used to bleach areas offluorophore within the cell. The microscope's light detectors may beturned off during the bleach phase, since the image data collection isnot required. After the bleach, a timed sequence of images is then takento observe transport of fluorophore back into the bleached region.

A simple bleaching experiment might include a root recording, whichdefines a scan over the whole cell, and one or more child recordings,which are the bleach regions. The data created when scanning the bleachregions is a kind of dummy, since the experimenter is often onlyinterested in controlling the amount of energy hitting the sample, andthe area over which illumination occurs. The user simply initiates atimed sequence of scans, upon which the root recording is scanned.Whilst the bleach region is being scanned, the user can toggle the“acquire” option on the bleach region, so that this region is onlyscanned once in the timed sequence.

The bleach region may be defined with exactly the same parameters as allother recordings, so it is guaranteed that the user has preciseinformation about the shape and position of the bleach region, as wellas input energy and the precise time at which the bleach occurred.

b) Photoactivation (Uncaging)

This is basically the opposite of photobleaching, whereby light is usedto activate a fluorophore. The parameters describing precisely when,where and how the compound was activated are neatly contained within arecording object.

c) Inhibition

This is similar to photobleaching. Compounds can be caged, or theirbiological function switched off by illumination with a characteristicwavelength.

Example 7

“Observation-only” Processes

In some experiments, samples can show Bioluminescence(Chemiluminescence), i.e. samples emit light without the need forillumination light.

1. A method of controlling a microscope with a computer comprising theuse of a recordings hierarchy of parent recordings and child recordings,each of which comprises a set of control parameter data, and of whichthe child recordings are linked to a single parent recording such that aselected group of control parameter data of the parent recording areinheritable by the child recordings, wherein the set of controlparameter data comprises a parameter indicating whether a recording isenabled or disabled in the recordings hierarchy.
 2. A method ofcontrolling a microscope with a computer comprising the use of arecordings hierarchy of parent recordings and child recordings, each ofwhich comprises a set of control parameter data, and of which the childrecordings are linked to a single parent recording such that a selectedgroup of control parameter data of the parent recording are inheritableby the child recordings, wherein the method comprises a step of savingrecordings from a computer memory onto a storage device such thatwhenever the parent recording is saved a selected group of the childrecordings are saved as well.
 3. A method of controlling a microscopewith a computer comprising the use of a recordings hierarchy of parentrecordings and child recordings, each of which comprises a set ofcontrol parameter data, and of which the child recordings are linked toa single parent recording such that a selected group of controlparameter data of the parent recording are inheritable by the childrecordings, wherein the method comprises a step of loading recordingsfrom a storage device into a computer memory such that whenever theparent recording is loaded a selected group of the child recordings areloaded as well.
 4. A computer controlled microscope comprising acomputer for controlling the microscope, wherein the computer is adaptedto execute a control program creating in a memory of the computer arecording hierarchy of parent recordings and child recordings, each ofwhich comprise a set of control parameter data, and of which the childrecordings are linked to a single parent recording such that selectedcontrol parameter data of the parent recording are inheritable by thechild recordings, wherein the set of control parameter data comprises aparameter indicating whether a recording is enabled or disabled in therecordings hierarchy.
 5. A computer controlled microscope comprising acomputer for controlling the microscope, wherein the computer is adaptedto execute a control program creating in a memory of the computer arecording hierarchy of parent recordings and child recordings, each ofwhich comprise a set of control parameter data, and of which the childrecordings are linked to a single parent recording such that selectedcontrol parameter data of the parent recording are inheritable by thechild recordings, wherein the computer is adapted to execute a controlprogram for saving recordings from a computer memory onto a storagedevice such that whenever the parent recording is saved a selected groupof the child recordings are saved as well.
 6. A computer controlledmicroscope comprising a computer for controlling the microscope, whereinthe computer is adapted to execute a control program creating in amemory of the computer a recording hierarchy of parent recordings andchild recordings, each of which comprise a set of control parameterdata, and of which the child recordings are linked to a single parentrecording such that selected control parameter data of the parentrecording are inheritable by the child recordings, wherein the computeris adapted to execute a control program for loading recordings from astorage device into a computer memory such that whenever the parentrecording is loaded a selected group of the child recordings are loadedas well.
 7. A data carrier carrying a computer software product to beloaded into a memory of a computer of or a computer linked to amicroscope for controlling at least one of the sequence of operationsand a status of said microscope, the computer software product havingcomputer readable code embodied therein which comprises means adapted tocreate in the memory of the computer a recordings hierarchy of parentrecordings and child recordings such that each recording comprises a setof control parameter data and such that each child recording is linkedto a single parent recording in such a way that control parameter dataof the parent recording are inheritable by the child recordings, whereinthe set of control parameter data comprises a parameter indicatingwhether a recording is enabled or disabled in said recordings hierarchy.8. A data carrier carrying a computer software product to be loaded intoa memory of a computer of or a computer linked to a microscope forcontrolling at least one of the sequence of operations and a status ofsaid microscope, the computer software product having computer readablecode embodied therein which comprises means adapted to create in thememory of the computer a recordings hierarchy of parent recordings andchild recordings such that each recording comprises a set of controlparameter data and such that each child recording is linked to a singleparent recording in such a way that control parameter data of the parentrecording are inheritable by the child recordings, wherein the computerreadable code comprises means adapted to save recordings from a computermemory onto a storage device such that whenever the parent recording issaved a selected group of the child recordings are saved as well.
 9. Adata carrier carrying a computer software product to be loaded into amemory of a computer of or a computer linked to a microscope forcontrolling at least one of the sequence of operations and a status ofsaid microscope, the computer software product having computer readablecode embodied therein which comprises means adapted to create in thememory of the computer a recordings hierarchy of parent recordings andchild recordings such that each recording comprises a set of controlparameter data and such that each child recording is linked to a singleparent recording in such a way that control parameter data of the parentrecording are inheritable by the child recordings, wherein the computerreadable code comprises means adapted to load recordings from a storagedevice into a computer memory such that whenever the parent recording isloaded a selected group of the child recordings are loaded as well.